Production Data Pipelines

Back to Home

01. Lesson Introduction
02. Extending Airflow with Plugins
03. Extending Airflow Hooks & Contrib
04. Demo 1: Operator Plugins
05. Exercise 1: Operator Plugins
06. Solution 1: Operator Plugins
07. Best Practices for Data Pipeline Steps - Task Boundaries
08. Demo 2: Task Boundaries
09. Exercise 2: Refactor a DAG
10. Solution 2: Refactor a DAG
11. Subdags: Introduction and When to Use Them
12. SubDAGs: Drawbacks of SubDAGs
13. Quiz: Subdags
14. Demo 3: SubDAGs
15. Exercise 3: SubDAGs
16. Solution 3: Subdags
17. Monitoring
18. Monitoring
19. Exercise 4: Building a Full DAG
20. Solution 4: Building a Full Pipeline
21. Conclusion
22. Additional Resources: Data Pipeline Orchestrators

Back to Home

13. Quiz: Subdags

SubDAGs

The diagram above shows two DAGs and their tasks. Which tasks could be placed into a SubDAG for reuse?

HTTP Data Fetch

Data Check

Merge

Check API Status

SOLUTION:

HTTP Data Fetch
Data Check

SubDAG Advantages

Which of the following are advantages of SubDAGs?

Improves code visibility

Reduces duplicated code in DAGs

Improves visibility into DAG steps in Airflow UI

Allows for simple reusability of commonly occurring task patterns

SOLUTION:

Reduces duplicated code in DAGs
Allows for simple reusability of commonly occurring task patterns

SubDAGs Disadvantages

Which of the following are drawbacks of SubDAGs?

Reduces visibility in the Airflow UI

Sharing of common patterns between DAGs

Harder to understand and maintain if improperly scoped

Encourages Premature Optimization

SOLUTION:

Reduces visibility in the Airflow UI
Harder to understand and maintain if improperly scoped
Encourages Premature Optimization

Learn Udacity: click here to learn more :)